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INTRODUCTION 



Training Evaluation : A Guide to Its Planning , 
Development, and Use in Agency Training 
Courses is the fourth in -a series being published by 
the Bureau of Training to examine various aspects 
of training technology and the application of an 
instructional systems approach to agency training 
programs. An instructional systems approach is 
simply a way of dealing with organizational train- 
ing requirements by accurately identifying the 
organization’s problems and needs, setting specific 
objectives, applying logic and analysis techniques 
to the problems, developing methods for their 
solution, and comparing measurement results with 
the established objectives . 1 The growing need for 
more effective and efficient training has prompted 
increasing numbers of training specialists and 
management officials to consider this analytical ap- 
proach in the design, development, and assessment 
of their training courses. 

When applied to the assessment of training 
courses, a systems approach can provide continu- 



ing feedback of trainee performance data for use 
by the training staff in judging the effectiveness of 
their training. In this context, the process by 
which these data are collected, analyzed, and in- 
terpreted is training evaluation. 

This paper describes one approach which can be 
used by the agency training specialist to assess 
the effectiveness of many of their internally devel- 
oped and conducted training courses. It offers a 
conceptual framework for evaluation; the essential 
steps in developing training objectives; an in- 
course training evaluation process; and a descrip- 
tion of the steps involved in developing and using 
data gathering instruments. The paper concludes 
with a discussion of the factors to be considered 
when implementing the kind of evaluation pro- 
gram described in this paper. However, the com- 
plexity of the total evaluation process precludes 
complete coverage here. Additional papers will be 
issued periodically supplementing and expanding 
this one. 



CONCEPTUAL FRAMEWORK FOR TRAINING COURSE EVALUATION 



At the end of a training course the training staff 
is faced with the question “Was this course suc- 
cessful?” As stated, the question is difficult to an- 
swer with any certainty. As with any complex 
event the answer to the larger question lies in the 
answers to a number of smaller questions. 

It is noted in Application of a Systems Ap- 
proach to Training : A Case Study that the even- 
tual purpose of training is to provide learning 
experiences which influence trainee behavior ac- 
cording to predetermined objectives . 2 In an ideal 
sense, the demonstration of the desired behavior by 
the trainee constitutes direct evidence that the 
course objectives have been met and the training 
was successful. However, it is often necessary for 



the training specialist to use less direct evidence 
when attempting to determine the success of a 
training course. For example, it is easier to “dem- 
onstrate desired behavior” in courses designed to 
lead to concrete skills such as typing, card punch- 
ing, and computation than in courses designed to 
lead to attitudinal changes, such as in supervisory 
and executive development courses. 

This difference in ease of evaluation is well 
known to training specialists and results from the 
desire to measure the extent to which objectives 
are achieved and to interpret the results in terms 
of some standards. The specificity with which the 
objectives and standards can be established or de- 
fined will in large measure affect the extent to 
which measurement can occur. In general, the 
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more specific they arc the easier it will be to evalu- 
ate and maintain control over the quality of the 
training product. 

Quality control describes a concept developed 
in industry to assure that the end-product con- 
forms to predetermined standards of quality, 
workmanship, size, and form. The basis of quality 
control, is a system of checks and inspections that 
will provide continuing feedback on the status of 
the manufactured product. Inspection occurs at 
critical points in the receipt, handling, and proc- 
essing of materials as they flow through the plant. 
Engineering drawings, materials, specifications, 
and other similar criteria are used as a basis for 
determining quality in the inspection process. 3 

The three steps necessary to establish a quality 
control system are : Identification of control areas ; 
determination of standards of quality; and sys- 
tematic evaluation to ascertain whether or not 
standards are being met. With appropriate modi- 
fications these steps are applicable to the training 
process, as shown below : 

Step I. The Identification of Control Areas . 

The term “control areas”, as used here, is identi- 
cal to the establishment of training objectives. 

Step 2. Determination of Standards of Quality . 

This is equivalent to the determination of what 
is to be considered acceptable achievement on 
the part of trainees: the standards of training 
performance. 



Step 3. Systematic Evaluation, 

This involves the determination of whether 
standards are being met, which equates to (1) con- 
struction of instruments for assessing trainee prog- 
ress, (2) periodic administration of these 
instruments (3) analysis and interpretation of 
data obtained and (4) feedback of results to train- 
ing specialists and agency management for use in 
making needed course adjustments. 

The ultimate purpose of quality control is to 
insure — through early identification and prompt 
correction of deficiencies as they arise during the 
processing or training period — an end product 
(manufactured article or trained person) which 
meets a predetermined standard of quality or 
proficiency. In the case of a training course, such 
continuous monitoring is necessary because even 
the most thorough planning and design represents 
only a best estimate of what will happen when the 
course is actually conducted. 

This analogy, between industrial and training 
quality control, should not be carried too far. In 
the case of manufactured articles, the goal is to 
make them, insofar as possible, identical with a 
prototype. In the case of trainees, the goal is to 
bring each up to a predetermined minimum ac- 
ceptable achievement standard ; but beyond this, it 
would be expected that most trainees would exceed 
this minimum standard in varying degrees. The 
ideal of uniformity, therefore, would not apply in 
the training situation. 



DEVELOPING TRAINING OBJECTIVES AND STANDARDS 



Training objectives are goals to be reached as a 
result of the training course. Ideally, they should 
be stated in performance terms, i.e., some actions 
or behaviors which the trainee is expected to ex- 
hibit as a result of his training experience. It is 
best to have both interim and terminal training 
objectives, because this allows for periodic in- 
course, as well as end-of-course, evaluation. 

The essential steps in formulating training ob- 
jectives include: 

— Determining what general improvements or 

changes are desired. 

— Selecting out from the Total improvements or 
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changes desired those which would best be 
learned in the training cours j. 

— Identifying the particular skills and knowl- 
edges which will lead to the desired performance. 
— Developing training objectives by describing 
the action or behavior expected of the trainee 
as a result of the training. 

When writing training objectives it is impor- 
tant to describe not only the action or behavior 
expected but also the conditions under which the 
behavior will be exhibited and the criterion, or 
standard, of acceptable training performance. For 
example, two training objectives illustrating these 
three elements might read : 
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Example 1. 

The trainee will develop a program in COBOL which 
will print an alphabetic listing c* those employee develop- 
ment specialists in the test deck who are at grade GS-11 
and above. (This describes the act to be performed.) The 
trainee will work independently and, with the aid of his 
notes and other reference material, complete his assign- 
ment within 1 class day. ( These are the conditions.) The 
trainee’s program will be judged satisfactory if it runs 
free of error after no more than three compilations. (This 
is the training performance standard.) 

Example 2. 

Given appropriate reference materials and the format 
guide, and with minimal supervision (These are the con- 
ditions.), the instructor-trainee will demonstrate his abil- 
ity to develop, in standard form, an instructor lesson plan 
suitable for a 30-minute classroom presentation. (This is 
the act.) The plan must be logical and properly detailed 
for platform use, and must be at least 00 percent correct 
as measured by the standard lesson plan checklist The 
trainee will be given up to two hours to complete this task. 
(This is the standard.) 

Since the subject matter of most courses is less 
precise than that represented by t/he two examples 
above, it is often difficult to state training objec- 
tives in such specific terms. Nevertheless, it is im- 
portant for evaluation purposes to write training 
objectives as specifically as the subject matter per- 
mits. Here are two examples of training objectives 
written for less precise subjects: 



Once the training objectives have been written 
and the standards of training performance estab- 
lished the next step in the development of an evalu- 
ation design is to select the approaches which seem 



Example 3. 

Using the seven fundamentals of position classification 
as taught during the course (These are the conditions .) , 
the trainee will demonstrate his skill at conducting a desk 
audit (This is the act.) which receives a rating of at least 
“Average” on the standard rating form completed by the 
instructor. (This is the standard.) 

Example 4. 

Given an employee relations problem in supervision, the 
trainee will demonstrate his ability to solve the problem 
using the six basic principles of problem solving taught 
in the course. (This is the act.) He will work independ- 
ently and may use class notes and other course references 
as desired, and may have up to 50 minutes to complete his 
work. (These are the eonditioyis.) To be considered ac- 
ceptable, the trainee’s solution must incorporate all six 
principles and result in a course of action which is judged 
feasible for implementation by the instructor. (This is the 
standard.) 

Writing training objectives in such specific lan- 
guage will require much time, study, and persever- 
ance. However, it is clear that when they are 
written in this form, subsequent tasks such as 
developing course content, selecting methods of 
presentation, and establishing criteria for quality 
control can be approached with considerably more 
confidence. 

EVALUATION PROCESS 

best fitted to the assessment of trainee progress. An 
in-course training evaluation process, incorporat- 
ing all three steps of the quality control system in 
sequence, is displayed below. 



AN IN-COURSE TRAINING 
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Figure 1 



STEP 1 Establishing Training Objectives 
STEP 2 Setting Standards of Training Performance 
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Feedback of Results to the Training Specialist for Use in Making Course Adjustments 
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In the figure above, STEP 1 represents those 
actions or behaviors which the trainees are ex- 
pected to exhibit. STEP 2 represents the minimum 
acceptable level of achievement for each training 
objective identified in STEP 1. Stated a little dif- 
ferently, STEP 1 represents the desired outcomes 
of the training course, and STEP 2 defines those 
points on the learning continuum at which the out- 
comes can be said to have been achieved. 

STEP 3, then, represents that part of the proc- 
ess through which the training specialist deter- 
mines, on a continuing basis, whether or not the 
trainees are meeting the standards established in 
STEP 2. 

It should be noted that STEP 3 does not operate 
independently, or after the fact; rather it is part 



of a continuous feedback loop which contains four 
major components : 

(1) An on-going training course designed to 
accomplish the objectives established in STEP 

i; 

(2) Continuous, systematic collection of data 
bearing on trainee achievement ; 

(3) Analysis and interpretation of data col- 
lected in terms of standards established in 
STEP 2; and 

(4) Course modifications as appropriate. 

Thus STEP 3 provides a continuing flow of in- 
course evaluative data to the training specialist 
about the status of each trainee. While the evalua- 
tive data received in STEP 3 is ordinarily used in 
making day-to-day adjustments in the on-going 
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program, there are circumstances where the ad- 
justments required go beyond that and bring into 
question the appropriateness of the objectives or 
standards. When this occurs the feedback loop 
would also connect with STEPS 1 and 2. 

Simply stated, in-course evaluation data can 
be classified into two types : 

(1) Data that provides direct — or concrete — 
evidence about the extent to which the trainees 
can exhibit the desired actions or behaviors 
stated in a training objective; and 

(2) Data that provides indirect — or inferen- 
tial — evidence about training objective achieve- 
ment. 

Neither of these types of data taken alone can 
serve as a completely satisfactory basis for course 
evaluation. Both are needed to help counteract the 
limitations that are present in each. By collecting 
both types of data the training specialist will be 
in a much better position to ascertain whether or 
not course objectives are being met. 

Direct Data About Trainee Achievement 

This can be generated either informally or for- 
mally. Informal approaches such as role playing, 
case studies, class discussion, and trainee answers 
to instructor questions can provide the training 
specialist with an indication of the extent to which 
a trainee has achieved a given training objective. 
In some courses they are the principal source of 
direct data. 

Nevertheless, when using informal methods it 
is often difficult to collect sufficiently reliable data 
about the progress of each trainee. To obtain such 
data it is necessary to use more formal methods of 
direct data collection, such as written tests or per- 
formance exercises. The principal advantage of 
direct evaluative data collected through formal 
sources is that it is the best source of concrete 
evidence about the progress each trainee has made 
toward achieving the training objectives. 

The process of collecting direct data through 
formal sources has several potential limitations: 

(1) The subject matter. may not easily lend 
itself to formal measurement. 

(2) Constructing direct measurement instru- 
ments is a difficult task requiring expertness that 
may not be readily available to the training spe- 
cialist. 

(3) Many trainees are embarrassed by “tests” 
and resent their use in training programs. 



While these limitations are not inconsequential, 
the offsetting value of this source of achievement 
data often makes it worth the effort to overcome 
the limitations. 

Indirect Data About Trainee Achievement 

This may be as informal as a trainee’s incidental 
remark after class or as formal as a survey of 
trainee opinion. 

Informal data collection goes on continuously 
at the instructor level and is useful for suggesting 
minor adjustments in program presentation and 
administration. However, since such data is diffi- 
cult to collect systematically and interpret with 
confidence, informal data can rarely provide con- 
clusive evidence about trainee achievement. 

As with direct data collection, the most reliable 
data is obtained through the use of formal meth- 
ods such as trainee opinion surveys and classroom 
observations. 

The advantage of trainee opinion data, when 
systematically collected and properly interpreted, 
derive from the fact that a trainee is usually pres- 
ent for the total course and is thus able to make 
personal judgments about several important 
aspects of it, such as : 

(1) The extent to which he believes he has 
achieved the training objectives. 

(2) Those factors which he believes were re- 
sponsible for objectives being incompletely 
achieved. 

(3) The applicability of what he has learned 
to what he believes he will need in the work 
situation. 

(4) Whether or not he believes that the bene- 
fits received were worth the time and effort 
required. 

In addition, such data provides a general sum- 
mary of opinion about the overall quality of the 
course and the trainees’ satisfaction with what 
they have learned, both of which can be com- 
pared with the opinions of other groups which 
have attended the same (or a similar) course. 

However, data gathered from opinion surveys 
has two limitations which restrict the use to which 
that data can be put : 

(1) Trainee opinions provide inferential 
rather than direct evidence about the extent to 
which training objectives have been met. 

(2) It is difficult to assess the reliability and 
validity of the opinions expressed. 
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A complementary method for collecting indirect 
data about trainee achievement is classroom ob- 
servation. Applying this method requires the use 
of observers who know the objectives and subject 
matter of the session being observed and are fa- 
miliar with good training practices. 

When systematically collected by qualified ob- 
servers, the most important advantage of class- 
room observation data is that it provides third- 
party evidence which can be used to verify data 
derived from other sources. 



The limitations of the classroom observation 
method arc that : 

(1) It represents only an individual’s point 
of view. The observer, no matter how well qual- 
ified, cannot assess the instructional experience 
through the eyes of the learner. 

(2) It may lack perspective. The observer who 
assesses an isolated part of a larger instructional 
block does not have firsthand knowledge of what 
occurred in prior or subsequent sessions. 



DEVELOPING AND USING DIRECT MEASUREMENT INSTRUMENTS 



Developing instruments for collecting data 
about trainee progress requires substantial invest- 
ments of time and effort on the part of the train- 
ing specialist. 

To be effective, such measuring instruments 
must possess two fundamental qualities : validity 
and reliability. 

Validity refers to the relevance of the instru- 
ment to the purpose for which it is being used. 
Does it in fact do what it is intended to do? 
Will results provided by the instrument serve 
as an accurate indicator of the extent to which 
established training goals have been achieved? 

Reliability refers to the dependability of the 
data. How much reliance can be put in it? If 
data about the same (or comparable) classes 
were to be collected successively, would the re- 
sults be substantially the same? If the instru- 
ment were to be used to gather data about two 
trainees who had achieved equally, would the 
results reflect this? 

While the importance of developing valid and 
reliable measurement instruments is apparent, 
good instruments in themselves do not assure ef- 
fective evaluation; slipshod use can detract sub- 
stantially from their value in the training quality 
control process. 

This section (and the one that follows) describe 
how to develop and use direct and indirect meas- 
urement instruments for collection of evaluative 
data. 



The Development of Direct Measurement 
Instruments 

This involves ten steps which, when followed by 
the training specialist, will facilitate the develop- 
ment of valid and reliable instruments. The same 
steps should be followed for the development of 
either written tests or performance exercises. 

(1) Decide what is to be measured. 

Accomplishing this step involves: (a) Select- 
ing those training objectives against which 
trainee progress can directly be measured; and 
(b) Relating each objective to the instructional 
material which is intended to produce the desired 
action or behavior. 

As noted earlier, this step is fundamental to the 
entire evaluation process. 

(2) Develop a measurement matrix. 

In simple terms, a measurement matrix is a 
worksheet which allows the training specialist to 
conveniently sketch out evaluation strategies which 
he believes will result in the most efficient and 
effective measurement of the extent to which train- 
ing objectives have been met by the trainees. A 
sample measurement matrix is shown in figure 2. 
The seven major headings are discussed in the nar- 
rative text which follows. When completed, a 
simple measurement matrix would resemble the 
one shown in figure 3. 



Figtjre 2 



TRAINING MEASUREMENT MATRIX 



Couise Title: 

Course Length: Time to be Devoted to Measurement: 



(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 


Training 

objectives 

✓ — "" — - 


Hours of 
instruction 


Learning 

levels 


Measurement 

approach 


Assigned 

weights 


Estimated 

time 

Requirements 


Remarks 



Column 1. Training objectives have previously been dis- 
cussed in detail and, as noted in step 1 above, they are 
fundamental to the entire evaluation process. 

Column 2. Hours of instruction. For convenience, total 
clockliours are normally rounded to the nearest whole 
hour. 

Column 3. Learning levels can be conveniently categor- 
ized under three headings : 

(a) Familiarization, where trainees learn sufficient 
facts and principles to be able to recognize their impor- 
tance and know where to go for additional knowledge 
should the need arise. 

(&) Working knowledge, where trainees possess a 
sufficient knowledge of related facts, principles, and 
techniques to enable them to perform routine practical 
applications under the direct supervision of a qualified 
individual. 

(c) Qualified, where the trainees are capable of 
demonstrating sufficient knowledge to permit independ- 
ent functioning in the area concerned. 

Column 4. Measurement approach. Basically, there are 
two approaches by which trainee achivement can be di- 
rectly measured. These are : 

( a) Written [bests, of which there are two main, types : 
the objective (true/false, multiple choice, matching, 
etc.) and the subjective (essay) ; and 

(b) Performance exercises, in which the trainee is 
required to perform the actual skills being taught under 
closely controlled conditions. Performance exercises are 
used frequently in the training of maintenance and 
service personnel. However, in recent years, such exer- 
cises have been used to train supervisors, counselors, and 
classifiers. 

Naturally the approach selected will have to be con- 
gruent with the learning level identified in column 4. 



Thus : 

• Measurement at the familiarization level would 
normally require trainees to recognize the correct 
facts and principles and suggests the use of objective 
(limited response) questions in the measuring 
instrument. 

• Measurement at the qualified level would normally 
require extensive use of performance exercises. 

From the above, it should not be inferred that there 
is an arbitrary relationship between the learning level 
and the type of approach used. Depending on the situa- 
tion, satisfactory measurement at the working knowl- 
edge level, for instance, may be accomplished using 
objective test items. The important thing to remember 
is that the measurement approach employed should be 
compatible with the learning level. 

Column 5. Assigned weights. Weights are assigned, in 
percentages, according to -the importance of each training 
objective. If the number of hours of instruction given to 
accomplish each course objective we^*e exactly equivalent 
to their importance, the weighting would be directly pro- 
portional. However, this is rarely the case. In most in- 
stances the amount of time required to bring about the 
desired objective will be either greater or less than the 
absolute weight (or importance) of that behavior in the 
total course. 

Column 6. Estimated time requirements are educated 
guesses about the amount of time which would be required 
to administer the measurement instrument selected in 
column 5. A tally of the time requirements listed in column 
6 for a given course can serve as a quick check on the 
feasibility of fitting the evaluation strategies selected to 
the course time schedule. 

Column 7. Remarks. This column conveniently allows the 
training specialist to “footnote” his decisions with ques- 
tions, qualifications, reminders, and suggestions. 
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(3) Develop the elements of the measurement 
instrument. 

The basic element of the written exercise is the 
test item. For the performance exercise the re- 
quirement would be to perform a required task. 
When developing measurement elements, several 
points should bo kept in mind : 

• The element employed should be compatible 
with the learning level as specified in the 
training objective statements. 

• Elements should require the application of the 
information learned rather than being limited 
to mere recall of facts. 

• Elements should be clearly stated so there is 
no chance of misinterpreting their intent, 

• Only elements which bear directly on the at- 
tainment of training objectives should be 
used. 

• Elements should range in difficulty from those 
which nearly all trainees may be expected to 
answer to those which challenge even the best 
trainees. 

(4) Assemble the measurement instrument. 

The following are generally accepted principles 
for arranging the elements of a measurement 
instrument : 

• If different kinds of elements are used, group 
like kinds together. 

• Insofar as possible, arrange the elements so 
that a few of the easier ones come first. This 
will give the trainee early success -and not 
frustrate him. 

( 5 ) Write directions for administering the 
instrument. 

Directions should be so complete that no addi- 
tional information need formally be provided be- 
fore, during, or after the exercise. In the case of a 
written exercise, the directions would appear on 
the cover sheet. 

(6) Dry run the instrument.' 

This is normally done by having several mem- 
bers of the staff take the test or work through the 
exercise, and then obtaining their opinions on such 
factors as validity, practicability, organization, 
level of difficulty, and length, 

(7) Establish the critical achievement point. 

This entails the determination of what repre- 
sents minimum acceptable achievement on the test 
or exercise. The establishment of the minimum 



satisfactory achievement level for a measuring in- 
strument is a decision made after consideration of 
all relevant factors, such as length of exercise, 
level of difficulty, and performance requirements. 
Scales used for indicating achievement range from 
very coarse two- point scales, such as “satisfac- 
tory 55 and “unsatisfactory 55 , to precise percentage 
scales (1 to 100). The scales employed would be 
dictated by the degrees of precision desired (or 
possible) in assessing achievement. 

( 8 ) Administer the instrument. 

If the directions are complete and clearly pre- 
sented, administration should go smoothly. If, on 
the other hand, trainees continually ask for clarifi- 
cation and additional information after the test or 
exercise is underway, some revision and/or expan- 
sion of directions is in order. 

( 9 ) Analyze and evaluate results of the 
instruments. 

Although the preceding steps do much to assure 
the development of a good measuring instrument, 
they do not guarantee a finished product. Certain 
inadequacies may have slipped by unnoticed. Now, 
after it has been administered, results should be 
analyzed and inadequacies identified, 

(10) Revise the instrument based on the above 
analysis. 

This revision should begin immediately after 
weaknesses have been identified and verified. 

Using Direct Measurement Instruments 

This involves a relatively straightforward pro- 
cedure consisting of three steps : Administer the 
instrument; process the results; and hold a 
critique. The following paragraphs describe how 
the steps should be applied to realize the full bene- 
fits from direct measurement instruments. 

(1) Administer the instrument. 

In order for a trainee to carry out a written or 
performance exercise, he must be fully informed 
about the parameters of the exercise. Directions 
should be clear and complete — the goal being to 
anticipate and answer those questions which a 
trainee might be expected to ask before the, exer- 
cise begins. Directions normally include informa- 
tion on the following : 

(a) Time considerations, including : 

• Total time allocated to the exercise as well 
as allocations for subelements of the exer- 
cise where applicable ; and 



• Advice to the trainees on how to use their 
time effectively. 

(6) Specific directions on how to carry out 
each different element in the exercise. 

(c) Whether or not the trainee will be per- 
mitted to ask questions during the exercise or 
make use of reference materials. 

(2) Process the results. 

It is important to score and analyze the data 
collected quickly while trainee interest is still cen- 
tered on the measured area of instruction. 

(a) Scoring . Where measurement is accom- 
plished by use of objective test items, accurate 
scoring presents no problem. Where other kinds 
of items arc employed (problem solving and 
essay, for instance) the element of judgment is 
introduced and scoring can become quite sub- 
jective unless specific criteria are established 
for assigning credit. This caution applies 
equally to performance exercises. 

(5) Analysis of results . This involves the 
graphic display of score distribution and the de- 
termination of mean, median, and modal scores. 
These data make possible a quick comparison of 
the achievement of this group with established 
norms (where an equivalent pretest has been 
used, the training specialist can also determine 
the progress which the trainees have made from 
their pretraining performance level). In addi- 



tion, an item analysis, which shows group 
achievement on the individual items or parts 
of the exercise, should be done to pinpoint areas 
of trainee difficulty. 

(3) Hold a critique. 

In keeping with the learning principle of “im- 
mediate confirmation”, a critique should be con- 
ducted immediately after the exercise has been 
scored and the results analyzed. A prolonged lapse 
of time between measurement and critique may 
cancel out the value of the latter as a teaching 
device. 

When conducting the critique, the training spe- 
cialist should concentrate on those parts of the 
test or exercise with which a significant portion 
of the class had difficulty. Seldom missed parts may 
be largely disregarded. To keep the critique from 
becoming a “face off” between the instructor (de- 
fending “school solutions” to elements selected for 
review) and individual trainees (who disagree 
with these solutions), a useful technique is to di- 
vide the class into committees and assign one or 
more of the exercise problems to each committee 
which then assumes the responsibility for reaching 
a consensus concerning the solution of their as- 
signed problems and for presenting their solutions 
to the class. Such a trainee-centered critique can 
provide a more dynamic learning situation for 
trainees, and solutions thus presented may be more 
credible to those who missed the problems. 



DEVELOPING AND USING INDIRECT MEASUREMENT INSTRUMENTS 



This section will describe how to develop and 
use two kinds of indirect measurement instru- 
ments: those designed to gather trainee opinion 
and those for collecting the opinions of qualified 
classroom observers. 

The two forms for indirect performance meas- 
urement described in this section are not meant 
to be prescriptive models. They are only sample 
forms which have been developed to capitalize 
on the values of opinion while minimizing the lim- 
itations of this source of information. 

In general, the same steps described earlier for 
the development of direct measurement instru- 
ments apply here, with two exceptions : there is no 
need to develop a measurement matrix and no need 
to establish critical achievement points. 



Developing a Traiuee Opinion Survey 

This involves the use of two types of questions : 
controlled or free response. 

A controlled response question is one in which 
the responses are provided and the trainee checks 
the one which best reflects his opinion. This type 
of item has the advantage of forcing trainee re- 
sponses toward specific issues. It also facilitates 
tabulation and analysis. The principal limitation 
to this approach is that the trainee must confine 
his response to the alternatives given and is not 
permitted to provide additional opinions which 
may be, in fact, more important than those asked 
by the survey designer. 

A free response item is one that poses an open- 
ended question which the trainee answers in his 
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own words. The advantages and limitations of free 
response items are just the reverse of those for 
limited response. 

• They are easy to develop, and allow for un- 
limited responses. 

• The responses may be difficult to tabulate and 
analyze, and they may not relate to significant 
issues. 

Typically, training specialists have found that 
a combination of the two types of items is prefer- 
able. 

Figure 4 shows the front side of a sample ques- 
tionnaire which solicits trainee opinion primarily 
in the area of objective attainment. The emphasis 
is, therefore, on instructional “effects” rather than 
causal factors. Opinions about these factors — in- 
structor qualities, training aids, facilities, etc. — is 
asked for only when the trainee feels they pre- 
vented the intended instructional “effect” from 
being fully attained. Thus, the approach used in 
this form differs from the traditional approach, 
where trainees are requested to rate directly the 
elements of course delivery (or causal factors) , and 
where, based on the resulting concensus, the trainer 
makes certain assumptions about the degree to 
which the desired instructional “effects” (or ob- 
jectives) were attained. 

Colurrm 1 in the figure is keyed to a list of 
training objectives. This provides the trainee with 
a precise frame of reference within which to base 
his opinions about the extent to which each objec- 
tive has been met. 

Colurrm 2 provides the trainee with an oppor- 
tunity to register his opinions about the extent 
to which he personally believes that he has 
achieved each objective. 

Colurrm 3 allows the trainee to indicate those 
factors in the course which he feels impeded his 
achieving a given training objective. 

Column 4 provides an opportunity for the 
trainee to look beyond the training situation to the 
work environment and make judgments about the 
degree to which he believes a given Objective 
relates to his job. In effect, the trainee is asked to 
make judgments about the validity of the train- 
ing objectives. (Where the training is designed to 
prepare him for a job which he has never per- 
formed, this column can be eliminated.) 

Colurrm 5 asks the trainee to indicate his opin- 
ion about the extent to which a given abjective 
was worth the time and money expended on it 



during the course. (As with column 4 above, this 
column should be dropped where the trainees have 
no basis for judgment.) 

The back of the form (not shown) can be used 
to solicit additional opinions of particular interest 
to the training specialist. These might include 
opinions about : 

• General matters, such as the significant 
strengths and weaknesses of a course or the 
amount of emphasis placed on particular cur- 
riculum areas; 

• Specific aspects of a course, such as the con- 
venience of the training site, length of ses- 
sions, or use of a particular instructional aid; 

• Personal changes in beliefs or attitude which 
do not lend themselves to performance meas- 
urement. 

The Use of Trainee Opinion Surveys 

This parallels the direct evaluation procedure 
with the exception of the critique, which is not 
applicable. 

(1) When should an opinion survey be given? 

Traditionally trainee comments have been 
sought at the end of a course. Survey question- 
naires are distributed on the last day to be either 
completed in class or taken home and returned 
later. The two main advantages of conducting the 
survey at this time are the ease of administration 
and the perspective possessed by the trainee who, 
having experienced the entire course, is able to 
comment on how effectively its parts fit together in 
light of the overall objectives. A limitation to the 
survey at the end of the course is that the trainee 
may concentrate his comments on the last part of 
the course (which he remembers more vividly) and 
overlook the first part. 

If the types of skills, knowledges, or attitudes 
being taught can be broken down into major course 
units, each governed by a performance objective, 
trainee opinion surveys can be administered at the 
end of each unit when the objectives and the asso- 
ciated instruction are fresh in the trainee’s mind. 

The decision as to which approach, a single end- 
of-course survey or several end-of-unit surveys, 
should be used depends largely on two factors. One 
factor is course length : If it is a short course, one 
survey at the end will likely suffice; while, if the 
course is long, it may be desirable to administer 
one or more interim end-of-unit surveys. The 
other factor relates to the number of times the 
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)N QUESTIONNAIRE 

ingful training experience, we would appreciate your opinion about the 
ectives established for this block of instruction. Attached is a list of 
he questions below by placing a check mark in the appropriate block, 
why in the space provided at the bottom and on the back of this sheet, 
comments. Attach extra sheets if additional writing space is required. 


(5) 

Do you believe the benefits 
derived from pursuing this 
objective were worth the time 
and effort? 


Not at 
all 

(explain) 

below) 
















To some 
extent 
(explain 
below) 
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(4) 

Do you believe that this 
objective was related to the 
requirements of your job? 


Not at 
all 

(explain 

below) 
















To some 
extent 
(explain 
below) 
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(3) 

If the objective was not at all, or only partially, achieved, 
what factors were responsible? (Please check and explain 
below) 
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(2) 

As far as you are con- 
cerned, was this objec- 
tive achieved? 


Not 

at 

all 
















To some 
extent 
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Explanation of factors checked above. 



course has been conducted : If it has already been 
given several times, a single, cnd-of-course survey 
may well be all that is required. If, on the other 
hand, it is a new course, the training specialist may 
want to administer several end-of-unit surveys as 
a means of gaining more specific trainee comments 
and suggestions that can be referred to in making 
subsequent course revisions. 

( 2 ) Who should give it? 

Because objectivity is important to the quality 
control effort, it is best to have the survey con- 
ducted by someone who has not been directly con- 
nected with the presentation of the course. This 
person, however, should be knowledgeable of 
course objectives as w T ell as familiar with the tac- 
tics and strategies used to present the course mate- 
rial. 

(3) How should it be given? 

Regardless of the time chosen to conduct the 
survey, the three basic functions of the adminis- 
trator are : 

• To explain the purpose of the survey, in order 
to avoid confusion and thus increase the likeli- 
hood of receiving relevant comments. 

• To provide an overview of the course and its 
objectives to help refresh the trainees’ 
memories. 

• To administer the survey. 

(4) What should be done with the results? 

Data, once received, should be promptly tabu- 
lated and analyzed by the trainer. In his analysis 
he should look primarily for trends rather than 
isolated opinion, and he should consider the nar- 
rative comments along with the ratings. Interpre- 
tation, including verification of findings by com- 
parison with data from other sources, is the final 
step in this process. 

Regardless of the care with which survey results 
are analyzed and interpreted, they can have only 
limited value unless they lead to program improve- 
ment. Therefore, once the commitment to adjust 
the course has been made, prompt follow-up action 
is essential for effective quality control. 

Developing a Classroom Observation Ap - 
proach 

This involves procedures parallel fto those used 
in developing the trainee opinion survey discussed 
above. Therefore, it must be designed to collect in- 
formation about the extent to which training ob- 
jectives are being met, and identify those factors 



in the instructional process which appear to im- 
pede or facilitate trainees’ attainment of the 
objectives. Figure 5 shows a sample classroom 
observation guide, the main parts of which are : 

1. Training objectives . These provide a frame 
of reference within which the observer will 
base his judgments about the extent of objec- 
tive attainment. 

2. Assessment . Here the observer indicates the 
extent to which the objectives appear to have 
been attained by the trainees. 

3. Checks on attainment . Here the observer is 
required to identify and judge the effective- 
ness of the checks made by the instructor 
during the session to monitor the extent to 
which training objectives are being attained, 

4. Factors influencing objective attainment . The 
observer is required to add specificity to the 
general assessment which he made in 2 above. 

The Use of the Classroom Observation 
Approach 

This differs from other methods in that the 
emphasis is on an individual observing the train- 
ing situation and making informed judgments 
about the extent to which the objectives are being 
met. If the observation is to contribute signif- 
icantly to the evaluation effort, the observer should 
have a sound background in the subject matter of 
the session he is monitoring, a precise knowledge 
of its training objectives, and an awareness of how 
this session ties into the entire instructional 
program. 

The observer’s presence during class should not 
be conspicuous. Therefore, the sample observation 
form shown (fig. 5) has been designed to allow 
the observer to check the appropriate blocks dur- 
ing the session and fill in the required explanation 
later. 

After the session, the observer should discuss 
with the instructor those factors which were be- 
lieved to have facilitated or impeded the attain- 
ment of training objectives. Where appropriate, 
the oral critique should be followed by a written 
report in which effective and ineffective aspects 
previously discussed are noted along with recom- 
mendations for improvement. Such reports assume 
additional importance after the trainee perform- 
ance data has been obtained, since performance 
data only indicates what has occurred while the 
observation reports along with opinion surveys 
can be used to explain why the trainees did or did 
not meet the objectives. 
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Figure 5 



CLASSROOM OBSERVATION GUIDE 

Course Session_ 

Time Date Instructor 



1 . Training objectives: 



2. Assessment: To what extent did training objectives appear to be attained? (Adverse factors should 
be explained in item 4 below.) 

a. □ Fully. (Minor deficiencies, if any.) 

b. □ Partially. (Some factors preventing full achievement.) 

c. □ Hardly at all. 



3, Checks on attainment: What steps did the instructor take to assure himself that training objec- 
tives were being attained? 



How effective were they?. 



4 . Factors influencing objective attainment: Factors that had an adverse effect on training 
objective attainment as indicated in 2 above. (Check any such factors and explain on the back of 
this sheet.) 



Factor Check 

A. Introduction to training session 

B. Content 

C. Clarity of explanations 

D. Instructor’s apparent knowledge of 

subject 

E. Subject matter organization 

F. Selection and use of training aids 

G. Instructor’s manner (tact, etc.) 

H. Instructor’s ability to maintain trainees’ 

interest 



Factor Check 

I. Instructor’s ability to direct discussion^ 

J. Appropriateness of material to group __ 

K. Time/material relationship : 

L. Individual trainee involvement 

M. Instructional methods used 

N. Reference materials 

O. Classroom facilities 

P. Tests and practical exercises 

Q. Other (specify) 



(Observer’s Name) 
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FACTORS THAT INFLUENCE THE DECISION TO EVALUATE 



Earlier in the paper a process for the evaluation 
of an agency’s internally developed training 
courses was presented. Eeadcrs are cautioned that 
it represents an ideal process and not one which 
should be considered completely applicable to all 
courses or to all parts of a particular course. The 
evaluation process should be considered as a con- 
tinuum which ranges from little evaluation (or 
informal feedback) to comprehensive evaluation 
(which would include all four data collection 
methods described in this paper — opinion surveys, 
class observations, written tests, and performance 
exercises). Naturally, the points along this con- 
tinuum relate to the number and kind of evalua- 
tion methods employed. The real question then 
becomes : How much evaluation does a particular 
course require? Or, to state it differently: What 
point on the continuum best represents its evalua- 
tion needs? 

There are three variables which agency officials 
should consider when trying to decide how much 
evaluation a particular course requires. One of 
them — course content — strongly determines what 
can be done. The other two — learning level and 
course costs — suggest what ought to be done. 

( 1 ) Course content* 

This is particularly influential on how much 
measurement can be applied. Typically, direct 
measurement cannot be accomplished with the 
same precision in human relations training, for 
example, as it can in courses that are more techni- 
cal in nature. 

( 2 ) The Learning Level of the Course . 

Is the purpose of the course essentialy to famil- 
iarize, provide a working knowledge, or qualify? 
A course designed to qualify trainees in a par- 
ticular skill requires a more complete evaluation 
than an orientation program designed to familiar- 
ize entering employees with an agency’s operating 
procedures. 

( 3 ) Course costs . 

In general,- the more costly the course, the more 
comprehensive its evaluation ought to be. Costs 
may be considered from two points of view: 



[a) Per unit (trainee) cost. Factors influ- 
encing this cost are: Number of trainees; 
course length; personnel, equipment, and fa- 
cilities requirements for conducting the 
course ; and trainee grade level. 

(b) Course frequency. Courses that are re- 
peated frequently may have a relatively low 
per unit training cost, but may represent con- 
siderable overall cost because of the large 
number of personnel trained. Such courses 
may deserve comprehensive evaluation. On 
the other hand, extensive evaluation of “one- 
time” courses is seldom justified. 

A consideration of these variables helps to give 
agency officials a picture of what kind of evalua- 
tion plan a particular course may need. Whether 
or not such a plan can be implemented is in- 
fluenced by two factors. 

One of these factors is the degree of control ex- 
ercised by the agency over course delivery. Where 
this control covers all elements — media, facilities 
and, in particular, the instructional staff — the level 
of evaluation may be set based on consideration of 
the above variables. However, where it is desirable 
to use frequent guest faculty to present the pro- 
gram, control over the delivery stage will be more 
limited. As a consequence, it will be more difficult 
to apply achievement measures. 

The second influencing factor will be the re- 
sources available to the agency. A full scale evalua- 
tion plan represents considerable time, money, and 
professional competency on the part of the train- 
ing personnel involved. Since increased reliability 
comes at a high price, it must be decided what 
point on the continuum constitutes an acceptable 
trade off between cost and reliability. This de- 
cision making is further complicated because it 
does not normally involve only a single course, 
but rather every internally developed course, 
making up the agency’s training program. Agency 
officials must therefore set priorities and assign 
resources based on their needs for information — 
information which will tell them whether or not 
a given training experience has accomplished what 
was intended. 
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